VALIDATING THE HEIRARCHY 
OF THE iSTARTSMART® ACADEMIC CONTENT 


Perry W. McManis, M.A. 
Hatch Early Learning 

Mark H. McManis, Ph.D. 
University of Texas 

May 2016 



Abstract 


The purpose of this analysis was to investigate the validity of skill groupings in an instructional 
technology learning system designed for use by children in early childhood education classrooms. A 
Principal Component Analysis was performed to measure the fit of 18 skill games to their 5 assigned 
groupings in the system, covering a range of academic school readiness literacy and math skills. Using 
data from the developer’s o nlin e reporting system; achievement in those 18 skills was analyzed from a 
selected group of 274 preschool children from the United States with a high degree of game -play. These 
outcomes were loaded onto factors selected using an eigenvalue evaluation in an effort to deter min e the 
validity of the system’s representation of those groupings as they exist in the broader educational context. 
Results indicate a robust fit between system hierarchy and pedagogical hierarchy, with strong initial 
loadings along the existing hierarchy, minimal deviation, and weak subsequent loadings. These results 
suggest that the existing system hierarchy is a good match for the pedagogy which it purports to represent, 
and in this manner is appropriate for its intended purpose. This lays the groundwork for a future 
Confirmatory Factor Analysis, and modifications to the system to create an even better fit between the 
learning system and pedagogy. (Contains 2 tables). 
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Introduction 


The use of instructional technology has grown in prevalence in early childhood classrooms, particularly in 
recent years (e.g., Simon, Nemeth, &McManis, 2013; Blackwell, Lauricella, & Wartella, 2014). Such 
technology has shown promise for improving academic school readiness in young children if designed 
around appropriate content (e.g., Guernsey, Levine, Chiong, & Sevems, 2012; Clements & Sarama, 

2003). 

The set of foundational literacy and math skills necessary for children to be successful once they enter 
formal schooling is supported by a large body of evidence (e.g., National Early Literacy Panel, 2008; 
National Research Council Committee on Early Childhood Mathematics, 2009). For foundational literacy 
skills, the National Early Literacy Panelhas identified alphabet knowledge, phonological awareness, 
phonological memory, rapid automatic naming of letters/digits and writing/writing name as the early 
literacy skills most important for school readiness, as these most strongly and consistently predict the 
subsequent conventional literacy skills of reading and writing. For foundational math skills, the National 
Research Council Committee on Early Childhood Mathematics identified number, including whole 
number, operations and relations, as well as geometry, spatial thinking and measurement as the most 
important skill areas in which young children should be able to show competency. 

It is therefore appropriate for developers of instructional technology learning systems designed for 
preschool children to include such content, as well as ensure that its delivery to young users occurs in a 
meaningful way. 


Study Context 

The iStartSmart® by Hatch (iSS) is a touch-based, computer-assisted learning system designed for 
preschoolers. The system presents a set of school readiness skills in the form of child-directed games. As 
part of the built-in hierarchy, the system groups game -play opportunities of these readiness skills into 
“Skill Levels”; tutorial (mechanics instruction), emerging, developing, developed and completed 
(maintenance). These levels are rolled up into “Skills” covering the content, which are organized further 
by grouping them into “Skill Families”, aligned to existing findings widely accepted within the field of 
early literacy and math (e.g., National Early Literacy Panel, 2008; National Research Council Committee 
on Early Childhood Mathematics, 2009). 

When a child plays on the system, the skills within each family are presented on a continuum of 
complexity. There are five Skill Fam ili es covering 18 individual skills. Grouped under Phonological 
Awareness are sentence segmenting, initial sounds, blending compound words, segmenting compound 
words and onset rime; Alphabet Knowledge is solely represented by letter recognition; Language 
Development holding language vocabulary (with a focus on position vocabulary), spatial skills and 
measurement; Logic and Reasoning having the skills of common shapes, sorting and patterning; and 
Numeric Operations covering counting foundations, numeral recognition, sequence counting, objects in a 
set, addition and subtraction. 
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While the grouping of these activities is well supported in other contexts, there remains a question as to 
the validity of the hierarchy as it is presented in the specific implementation within the iSS system. We 
were therefore interested in applying a statistical examination of the grouping of the skills into fa milie s to 
validate the pedagogical underpinnings of the organization and delivery of that content to children during 
game -play. We considered the following hypothesis: 

The iStartSmart Family groupings will maintain the original hierarchy of academic school 
readiness s kill s. 


Methods 

Sample 

In order to ensure adequate power to reliably classify the skills into families, the sample included the 274 
preschool children having maintained at least a developing level in the system across the 18 skills from 
the 2012-13 school year. These data were retrieved from the developer’s database underpinning its 
reporting system, which holds a record of game -play information over the entire school year period. 
Permission for use of this dataset for data analysis with the purpose of understanding and improving the 
system fell under a signed EULA (End User License Agreement) and therefore was exempt from IRB 
approval. 

Analysis 

For this study, a Principal Component Analysis (PCA) was the methodology selected. Particular 
consideration was given to the decision to use a principal component approach instead of a confirmatory 
factor analysis approach. While the skill family groupings had a reasonably strong basis in established 
pedagogy, the lack of existing quantitative analysis of those groupings implied that an exploratory 
approach would be most appropriate. 

While the system interface may discriminate between levels of mechanical aptitude and technological 
familiarity, mechanical tasks in the system are lim ited in their complexity and breadth (e.g., tapping 
icons), and therefore the impact of the variance related to these has a low likelihood of being correlated 
with individual factors. While the academic content being presented differs substantially between Skill 
Families, the same lim ited set of straightforward mechanical tasks is repeated throughout all Skills and 
Skill Fa milie s. Technological familiarity as an influence is precluded by the nature of the sample; the 
children selected for inclusion in the analysis each logged considerable individually tracked progress with 
the system and as such would have experiential knowledge of the games, as well as the built-in instruction 
designed for children who might have lim ited access to technology outside of the classroom. 

While it may be reasonable to consider the existing hierarchy as a priori factors, the lack of any direct 
quantitative evidence to that effect, in combination with the high likelihood that the variance associated 
with those factors is shared across all of the factors, made it desirable for the statistical method to assign 
all of the variance of the included factors for exploratory purposes. A PCA by its nature imposes those 
constraints (Suhr, 2005). 
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A PCA generated a set of factors grouping the skills and then loading them onto each included factor, 
producing weights quantitatively estimating correlations between factors and variables. Weights can 
range in absolute value from 0 to 1, with higher weights indicating that the skill is more closely affiliated 
with that factor. The most commonly accepted way to measure the strength of a factor is by evaluating the 
eigenvalue of that factor. Although there are a variety of methods for deciding which factors are strong 
enough to keep in a final model, one commonly accepted method is to keep all factors with an eigenvalue 
greater than 1 (Kaiser, 1991); which was the method employed in this study. Those factors are then 
retained in the final model to which a rotation is applied; and the degree to which a skill can be ascribed 
to the fundamental characteristic represented by that factor can be estimated. 

Results 

From this analysis, the six components with an eigenvalue greater than 1 were retained in the final set. As 
shown in Table 1, the initial eigenvalues showed that the first factor accounted for 20.9% of the total 
variance, the second factor 13.5%, the third 9.5%, the fourth 9.0%, the fifth 6.5% and the sixth 5.5%; for 
a total of 65. 1%. Because of the starkness of difference in the fundamental skills measured by each 
family, an orthogonal rotation was applied to the factor loadings. The rotation chosen was a Varimax 
rotation with Kaiser normalization (SPSS version 21). With consideration to the eigenvalue inspection, 
this rotated PCA yielded the six uncorrelated factor solution presented in Table 2. 

Table 1. Eigenvalues for all components and kept components from iStartSmart skills using principal 
component analysis (/V=274). 


Total Variance Explained 


Component 

Initial Eigenvalues 

Extraction Sums of Squared Loadings 

Total 

% of Variance 

Cumulative % 

Total 

% of Variance 

Cumulative % 

'l 

3.769 

20.940 

20.940 

3.769 

20.940 

20.940 

r 2 

2.440 

13.558 

34.498 

2.440 

13.558 

34.498 

r 3 

1.711 

9.503 

44.001 

1.711 

9.503 

44.001 

r 4 

1.626 

9.033 

53.034 

1.626 

9.033 

53.034 

r 5 

1.160 

6.446 

59.480 

1.160 

6.446 

59.480 

r 6 

1.010 

5.612 

65.092 

1.010 

5.612 

65.092 

r 7 

.875 

4.860 

69.952 




r 8 

.714 

3.967 

73.920 




r 9 

.670 

3.722 

77.641 




r io 

.594 

3.301 

80.943 




r n 

.587 

3.263 

84.205 




r 12 

.555 

3.085 

87.291 




r 13 

.477 

2.649 

89.940 




r 14 

.453 

2.517 

92.457 




r 15 

.424 

2.356 

94.813 




'l 6 

.387 

2.149 

96.962 




r 17 

.343 

1.907 

98.869 




r 18 

.204 

1.131 

100.000 





Extraction Method: Principal Component Analysis. 
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Table 2. Factor loadings from iStartSmart skills using principal component analysis with varimax 
rotation. 


Rotated Component Matrix 


Skill 

Skill Family 

Component 

1 

r 

2 

r 

3 

r 

4 

r 

5 

r 

6 

Sentence Segmenting 

Phonological Awareness 

.524 






Initial Sounds 

Phonological Awareness 

.697 






Blending Compound Words 

Phonological Awareness 

.797 






Segmenting Compound Words 

Phonological Awareness 

.859 






Onset Rime 

Phonological Awareness 

.825 






Counting Foundations 

Numeric Operations 





.904 


Numeral Recognition 

Numeric Operations 





.608 


Sequence Counting 

Numeric Operations 


.636 





Objects in a Set 

Numeric Operations 


.806 





Addition 

Numeric Operations 


.788 





Subtraction 

Numeric Operations 


.659 





Language Vocabulary 

Language Development 



.782 




Spatial Skills 

Language Development 



.791 




Measurement 

Language Development 



.836 




Letter Recognition 

Alphabet Knowledge 






.787 

Common Shapes 

Logic and Reasoning 




.764 



Sorting 

Logic and Reasoning 




.805 



Patterning 

Logic and Reasoning 




.668 




Extraction Method: Principal ComponentAnalysis. Rotation Method: Vhrimax with Kaiser Normalization, 
a. Rotation converged in 6 iterations. Values under .4 suppressed 


Discussion 

The loading of the factors matched the learning system’s current organization well, with four of the six 
components mapping directly onto the existing skill families. All of the skills grouped in the system under 
Phonological Awareness, Language Development, Logic and Reasoning, and Alphabet Knowledge 
loaded onto single factors; factors 1, 3, 4 and 6, respectively. 

While the loadings for the skills in the Numeric Operations family split onto two factors (Factors 2 and 
5), the nature of that split fits well with that of the inputs. Counting Foundations and Numeral 
Recognition both loaded onto a single factor (Factor 5), and both of those skills are recognition skills. 

The skills that loaded onto Factor 2 were Sequence Counting, Objects in a Set, Addition and Subtraction; 
each requiring the application of a recognized object to an operational task. In conjunction with the game 
design differences that split neatly along those lines (the skills loading onto factor 5 being prerequisites to 
those loading onto factor 2), while the skills would ideally load onto a single factor, indicating a single 
fundamental driver to line up with the system’s grouping, there is a reasonable enough fit to justify 
placing these under the Numeric Operations family in the system to remain w ithin the bounds of sound 
pedagogical theory. Given that these skills are all presenting mathematical concepts, these two factors fit 
well within the family for presentation purposes to early learners. 
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Additionally, the strength of the loadings was high, with no skill having a highest loading under .52, with 
all but one of the 18 meeting the convention of loadings of over .6 for high; and with none falling below 
the .4 guideline for low loading (Hair, Black, Babin, Anderson, & Tatham, 2006). Furthermore, 
subsequent loadings were weak, giving additional credence to the fit of the skills to their fa milie s. While 
this analysis suggests a good fit between the system and its pedagogical underpinnings, there do remain 
concerns regarding the limitations that arise around sample size. With a variable to factor ratio of 3:1, a 
larger dataset would have been ideal; especially given the exploratory nature of the present work. 

Conclusions 

The loading of the factors fits well with the pedagogical understanding of academic skill acquisition and 
its hierarchy. The differences between the theoretical and the computational skill loadings are reasonably 
explained by existing pedagogical theory. There is then a strong likelihood that there exists adequate 
evidence to support our hypothesis that the iStartSmart Family groupings would maintain the original 
hierarchy of academic school readiness skills. These results strongly suggest the manner in which the 
literacy and math content is organized and presented within the system has both a pedagogically and 
psychometrically sound basis. This lends support for the use of the system as an appropriate choice of 
instructional technology for promoting academic school readiness. 

Future Directions 

The results, when considered in light of the lim itations of the analyses, suggest value in performing a 
confirmatory factor analysis in the future with a larger dataset. As the iSS learning system sees a more 
widespread deployment, there should be a considerably larger set of data to work with in future analyses. 
It might also be of interest to test in a limited setting what effects splitting the Numeric Operations sk ills 
into distinct fa milie s might have on the factor loading as well as on level achieved by children. If that 
were to show a positive effect on level achieved, that could drive change in the future development of the 
system. 
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